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SUMMARY 


This report describes an effort which investigated concepts for 
displaying dynamic system status and fault history (propagation) 
information to the flight crew. This investigation was performed by 
developing several candidate display formats and then conducting 
comprehension tests to determine those characteristics that made one format 
preferable to another for presenting this type of information. Twelve 
subjects participated. Flash tests, or limited time exposure tests, were 
used to determine the subjects' comprehension of the information presented 
in the display formats. It was concluded from the results of the 
comprehension tests that pictographs were more comprehensible than both 
block diagrams and text for presenting dynamic system status and fault 
history information, and that pictographs were preferred over both block 
diagrams and text. It was also concluded that the addition of this type of 
information in the cockpit would help the crew remain aware of the status 
of their aircraft. 


INTRODUCTION 


Problem Statement 

Advances in the technologies of Artificial Intelligence (AI) software 
and Cathode Ray Tube (CRT) cockpit instrumentation hardware have produced a 
unique situation. New and potentially useful types of information are 
capable of being generated by a fault monitoring and diagnosis system 
connected to the aircraft (ref. 1), and electromechanical dials and gauges 
are no longer the only media available for displaying information to the 
flight crew. However, the problem remains of exactly how to present the 
new types of information generated by advanced diagnostic systems. 

Efforts have been made to design display formats for cockpit CRTs to 
convey system status information to the flight crew (refs. 2,3). But these 
new formats are still limited to displaying the current state of the 
aircraft. No provision is made for displaying the dynamics of fault 
propagation through aircraft systems, nor for taking into account the 
uncertainty associated with sensor input. 


Rackaround 

A research effort is currently underway in the Vehicle Operations 
Research Branch at NASA Langley Research Center to investigate the 
application of AI to flight management for improved safety and efficiency. 
This effort is focused primarily on civil transport aircraft operations. 
Several application areas exist within this focus that may benefit from AI 
including: fault monitoring and diagnosis, planning and replanning, and 

communications management (ref. 4). An application presently being 
explored is automated fault monitoring and diagnosis, with an emphasis on 
crew aiding and decision support . 

The flight domain presents many problems concerning the use of 
established AI technology, such as expert systems, for fault diagnosis. 


The most obvious challenge is that the situations are dynamic; that is, the 
status of the aircraft is constantly changing. Facts known to be true at a 
given time cannot be assumed to be true later in time. Also, sensors are 
installed only on a limited number of aircraft components, so diagnosis 
must often be performed with incomplete information. Thus, the dynamic and 
uncertain nature of the flight domain requires a new approach to fault 
monitoring and diagnosis that accounts for these factors. 

An automated fault monitoring and diagnosis concept that takes into 
account the dynamic and uncertain nature of the flight domain has been 
developed in the Vehicle Operations Research Branch. The general framework 
for this concept, as shown in figure 1, includes several distinct 
functions : a fault monitoring function, a fault diagnosis function, a 

recovery planning function, and an interface function (ref. 1) . 



Figure 1. General Framework for Automated Fault Monitoring and Diagnosis. 


The fault monitor analyzes aircraft sensor data to determine if an 
abnormal situation exists. The fault diagnosis function determines the 
cause or source of the problem, the affected aircraft components, and the 
propagation path of the fault through the aircraft systems. The response 
planner analyzes the diagnosis and determines the best course of action to 
compensate for the problem, and the interface function relays this system 
status, diagnosis, and response information to the flight crew. 
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Th© specific objective of the research described in this report was to 
support the development of the interface mechanism, in the above framework, 
by investigating concepts for displaying dynamic system status and fault 
history (propagation) information to the flight crew. This investigation 
was performed by developing several candidate display formats and then 
conducting comprehension tests to determine those characteristics that made 
one format preferable to another for presenting this type of information. 

It should be noted that the display of dynamic system status and fault 
information is only one of many research issues involved in presenting 
fault monitoring and diagnosis information. Other issues include: methods 
for presenting alternative hypotheses, prioritizing hypotheses based on 
some measure of likelihood, providing an explanation capability of the 
process used to reach a particular conclusion, and displaying the actual 
cause of a fault. All these issues are important, but their consideration 
was beyond the scope of this report. 


TECHNICAL APPROACH 


The following approach was taken to investigate concepts for the 
display of dynamic system status and propagation information. First, the 
general information requirements of the flight crew were identified with 
regard to a major aircraft system. The system chosen for this study was a 
generic turbofan engine. Once these crew information requirements were 
identified, display format concepts were developed to fulfill them. A 
generalized fault display interface program was then designed and 
implemented to provide the hardware display of these format concepts. 

Next, a series of tests was conducted to measure subject comprehension of 
the information presented in the different formats. The results of these 
tests were then analyzed to determine the display format characteristics 
that resulted in the best comprehension. 


Information Requirements 

The first step in the approach was to identify the information 
requirements of the flight crew with regard to fault diagnosis and system 
status. These information requirements were derived from the results of 
previously conducted interviews with experienced airline and test pilots. 
From the results of these interviews (ref. 4), the information requirements 
of the crew can be summarized as follows: 

(1) the crew wants to know what systems have failed or are affected by a 
failure; 

(2) they want to know what systems are likely to fail in the near future; 
and 

(3) they want to know what the severity of the failure is (what 
capabilities remain) . 
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Display Formats. 


Two distinct types of display formats were identified as being 
necessary to fulfill the crew information requirements . The first type is 
a system status display, which shows the components affected by a failure 
and the severity of the component failures. Since a system status display 
will include all major components of a particular system, the remaining 
capability of that system may be inferred by noting which components are 
unaffected. The second type of display format is a fault history display, 
which shows a summary of the changes in the status of the system 
components. The propagation path of a fault through a particular system is 
detected from this fault history display, which allows the crew to infer 
which components may be affected next. 

For the system status type of display, two different display formats 
were developed. The first format used pictographs to represent the system 
(fig. 2) . A pictograph is a graphical representation of a component or 
system that depicts its actual physical appearance or function. The second 
system status display format used block diagrams to represent the system 
(fig. 3) . A block diagram represents the functional relationship between 
components in a system, but does not necessarily depict the actual physical 
appearance or function of that system. 
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Figure 2. Pictograph System Status Format. 
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Figure 3 . Block Diagram System Status Format. 



Figure 4 . Pictograph Frames Fault History Format. 
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SUBSYSTEM NAME STATUS 

TIME 

left fan 

failed 

10:18:25 

left_combustor 
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10:18:26 

right_compressor 
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10:18:28 

right_turbine 

failed 

10:18:29 
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Figure 6. Textual Fault History Format. 
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For the fault history type of display, three different display formats 
were developed. The first two of these formats used four fault frames; 
each frame is a snapshot of the system status taken every time a change in 
status of one or more system components occurred. The four frames are 
shown on the screen simultaneously, and show the recent history of the 
faults in that system. The first fault history display format used 
pictographs to represent the system in the fault frames (fig. 4), while the 
second fault history display format used block diagrams (fig. 5). The 
third fault history display format used text (fig. 6). Here, instead of 
graphically representing the system using pictographs or block diagrams, 
the names of the components, their new status, and the time of change in 
status were listed in chronological order. 

In the display formats that used pictographs or block diagrams, color 
was used to differentiate between components with different status. Red 
was used to indicate the total failure of a component, amber was used to 
indicate a partial failure of a component (that is, a component that is 
still operating but only at reduced capability) , and no color to indicate a 
fully operational component. The outlines of all component diagrams were 
drawn in green. The textual fault history display format, however, did not 
use color. Instead, the new status of each component was listed with the 
component name and the time of the change in status. 

The general form of the pictograph formats was developed in accordance 
with the format guidelines prescribed by Summers and Erickson (ref. 2), 
except that fully operational components were not shaded green. Concepts 
were also borrowed from Way, Hornsby, et al. (ref. 3), regarding the screen 
layout of the pictograph engine display format. 


EaiiltJQi splay 


A fault display interface program was designed and implemented to 
generate the various display formats and to display on them the information 
generated by the fault diagnosis system. It also provided user interaction 
capability in the form of graphic menus for selecting the system status or 
fault history display format screens of different aircraft systems. Since 
this investigation was conducted independently of . the . development of the 
fault diagnosis function in the general fault monitoring and diagnosis 
framework, simulated fault diagnoses were provided by using multiple data 
files. However, the source of the fault diagnosis information was 
transparent to the test subject. 

The program itself was designed to be independent of the system being 
represented and the display formats being used. This was accomplished by 
storing, in separate data files, all the system hierarchy and format, 
(screen layout) information for the different display . formats (see fig. 7). 
The flexibility inherent in the design of the fault display interface 
program makes it useful as a tool for developing future display formats. 

The fault display interface program was implemented in Pascal on an 
IBM _ PCf using the TurboPascal programming environment (ref. 5). A Color 
Graphics Adapter (CGA) card, with 320 x 200 pixel screen resolution, and 
compatible color monitor were used to display the program output. 
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Figure 7. Fault Display Interface Program Schematic. 


EXPERIMENT DESIGN AND PROCEDURE 


Test Design 

To analyze subject comprehension of the information in the two types 
of display formats, two separate experiments were conducted. The first was 
a single-factor two-level experiment which compared the two system status 
display formats: pictographs and block diagrams (see fig. 8). The second 

was a single-factor three-level experiment which compared the three fault 



Pictographs Block Diagrams 

System Status Display Format 

Figure 8. Block Diagram of System Status Display Experimental Design. 
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Pictograph Block Diagram Textual 
Frames Frames 

Fault History Display Format 



Figure 9. Block Diagram of Fault History Display Experimental Design. 

history display formats: pictograph frames, block diagram frames, and the 
textual format {see fig. 9) . The ordering of the testing sequence in each 
of the two experiments was counterbalanced to prevent ordering bias and 
reduce the effects of learning (see Table 1) . Each display format in the 
two experiments was tested a total of 36 times. 


Subject 

Format Testing Sequence 

Experiment 1 

Experiment 2 

1 

B1,B2,B3,P1,P2,P3 

HB1 .HB2.HB3.HP1 ,HP2,HP3,HT 1 .HT2.HT3 

2 

P1,P2,P3,B1,B2,B3 

HT1 .HT2.HT3.HB1 .HB2.HB3.HP1 .HP2.HP3 

3 

B1,B2,B3,P1,P2,P3 

HP1 .HP2.HP3.HT1 .HT2.HT3.HB1 .HB2.HB3 

4 

PI ,P2,P3,B1 ,B2,B3 

HB1 .HB2.HB3.HT1 .HT2.HT3.HP1 .HP2.HP3 

5 

B1 ,B2,B3,P1 .P2.P3 

HP1 .HP2.HP3.HB1 .HB2.HB3.HT1 .HT2.HT3 

6 

PI ,P2,P3,B1 ,B2,B3 

HT1 .HT2.HT3.HP1 .HP2.HP3.HB1 .HB2.HB3 

7 

PI ,P2,P3,B1 ,B2,B3 

HB1 .HB2.HB3.HP1 .HP2.HP3.HT1 .HT2.HT3 

8 

B1 ,B2,B3,P1 .P2.P3 

HT1 .HT2.HT3.HB1 .HB2.HB3.HP1 .HP2.HP3 

9 

PI ,P2,P3,B1 ,B2,B3 

HP1 .HP2.HP3.HT1 ,HT2,HT3,HB1 .HB2.HB3 

10 

B1 ,B2,B3,P1 .P2.P3 

HB1 .HB2.HB3.HT1 ,HT2,HT3,HP1 .HP2.HP3 

11 

PI .P2.P3.B1 ,B2,B3 

HP1 .HP2.HP3.HB1 .HB2.HB3.HT1 .HT2.HT3 

12 

B1,B2,B3,P1,P2,P3 

HT1 .HT2.HT3.HP1 .HP2.HP3.HB1 .HB2.HB3 


Legend: 

Bx, Px 35 Displays using Block Diagram or Pictograph format 
HBx, HPx, HTx = Displays using Block Diagram Frame, Pictograph frame, 
or Textual history format 

Table 1. Test Sequence Ordering for the Various Formats Tested. 
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Te st Subjects 


Twelve test subjects participated in the experiments. Since no 
specific piloting tasks were needed for this test, a combination of pilots 
and non-pilots was used. Seven of the twelve test subjects were licensed 
pilots, with qualif ieations ranging from a Private Pilot License with 95 
hours total flight time to an Airline Transport Pilot (ATP) License with 
over 8,000 hours total flight time (see Table 2) . Of the other five 
subjects who were not pilots, four had experience in flight simulators, and 
all five were generally familiar with aircraft systems and operating 
procedures. Because of the small number and, in some cases, limited 
backgrounds of the test subjects in this study, the results cannot 
necessarily be generalized to the much larger population of operational 
pilots. However, the intent of this research was to investigate new 
concepts for displaying dynamic system status and fault history 
(propagation) information, and not to test specific formats that are 
candidates for imminent operational use. 


Subject 

Pilot 

License & 
Ratings 

Flight 

Time 

Flown 

Simulator 

Simulator 

Time 

Familiar 

w/Aircraft 

1 

No 



Yes 

1 

Yes 

2 

Yes 

Com/Ins 

450 

Yes 

50 

Yes 

3 

Yes 

ATP 

2700 

Yes 

1000 

Yes 

4 

Yes 

Private 

95 

Yes 

2 

Yes 

5 

No 



No 


Yes 

6 

No 



Yes 

120 

Yes 

7 

No 



Yes 

7 

Yes 

8 

Yes 

ATP 

8000 

Yes 

2000 

Yes 

9 

Yes 

Private 

300 

No 


Yes 

10 

Yes 

Private 

100 

Yes 

200 

Yes 

11 

No 



Yes 

15 

Yes 

12 

Yes 

Private 

4000 

Yes 

110 

Yes 


Legend : 

PRIVATE = Private Pilot’s License 
COM = Commercial Pilot’s License 
ATP = Air Transport Pilot’s License 
INS = Instrument Rating 

Table 2. Summary of Test Subjects’ Experience 


The test procedure for each subject is shown in Table 3. The major 
steps in the procedure are described in more detail in the following 
paragraphs, and all written test materials are included in Appendix A. 
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Each subject first received a short briefing on the overall research 
effort and the purpose of this particular series of experiments. The 
subject then filled out a background information questionnaire to record 
his pilot qualifications or aircraft-related experience. Also, before the 
tests began, each subject completed a questionnaire to record his ideas of 
what information would be useful from an automated fault diagnosis system, 
and how that information should be presented. This questionnaire was 
filled out by each subject before he had seen any of the display formats to 
be tested in the experiments. 

The display format comprehension tests were then conducted. Each test 
consisted of a series of flash tests for each display format. Each flash 
test, or limited time exposure test, was performed by presenting the 
display format for a short, measured amount of time on the CRT screen. 
Immediately after the image was erased, the test subject filled out a 
questionnaire (designed for that display format type) to record his. 
comprehension of the information presented. A separate and simplified 
version of the fault display interface program was used to generate the 
displays for the flash tests. 


Step 

Description 

1 

Introduction to research effort and current experiment. 

2 

Background information questionnaire. 

3 

Questionnaire prior to tests. 

4 

Discuss first system status format. 

5 

Testl, test2, test3. 

6 

Discuss second system status format (handout). 

7 

Test4, test5, test6. 

8 

Discuss first fault history format (handout). 

9 

Test7, test8, test9. 

10 

Discuss second fault history format (handout). 

11 

Testl 0, testl 1 , testl 2. 

12 

Discuss third fault history format (handout). 

13 

Testl 3, testl 4, testl 5. 

14 

Detailed introduction to fault display interface program. 

15 

Scenariol, scenario2. 

16 

Subjective evaluation. 


Table 3. Steps in Testing Procedure for Each Subject. 


Three flash tests were performed for each of the two system status 
display formats (pictographs and block diagrams) for the first experiment. 
Each of the three flash tests presented a different system status on the 
display format being tested, and the duration of each exposure was 1 
second. Before being tested on each format, the subject was allowed to 
study the format layout. Three flash tests were also performed for each of 
the three fault history display formats (pictograph frames, block diagram 
frames, and text) for the second experiment. Each of the three flash tests 
presented a different final system status and fault propagation path on the 
display format being tested. Because of the increased amount of 


11 




information on the fault history formats, the duration of the exposure was 
lengthened to 5 seconds. Again, the subject was allowed to study each 
format layout before being tested on it. 

After all flash tests were completed for each subject, the subject 
then participated in an interactive session with the fault display 
Interface program. During this session, the program g enerated a dynamic 
fault scenario in fast-time simulation (the frequency of faults was much 
higher than would normally be expected) . The fault scenario itself was 
designed to demonstrate the interactive capability of the fault display 
interface program, and was not meant to necessarily represent any real or 
expected series of events. No experimental data was collected from these 
interactive sessions; rather, they were included in this study to elicit 
subjective observations from the subjects about the interface mechanism 
concepts . 

At the conclusion of all the tests, each subject completed a 
subjective evaluation in the form of a structured questionnaire. This 
evaluation consisted of three parts. The first part contained seven 
multiple choice questions covering the operation of the fault display 
interface program, the two system status and three fault history display 
formats, and the usefulness of fault history information. The second part 
contained two short-answer/essay questions asking how the test subject 
would improve the display formats and the menu interface of the fault 
display interface program. The final part contained a single question 
asking the subject to rank the three features liked best and the three 
features liked least about the display formats. 


RESULTS AND DISCUSSION 


Flash Testa 


System. Sta tus Display., 


The tabulated results of the 


first experiment are shown in Tables 4a and 4b. The results show that 
comprehension was high for both display formats: 80 percent for the 
pictograph format and 88 percent for the block diagram format. The results 
also show that the test scores for the block diagram format were on average 
10 percent better than those for the pictograph format. When statistical 
analysis was applied to these results, however, it was found that the 
difference in scores was not statistically significant. The statistical 


analysis technique applied was the t-test (ref. 6). 


Format 

Comprehension (Average Test Score) 

Pictographs 

80.0 % 

Block Diagrams 

88.1 % 


4a. Test Results of System Status Display Format Experiment. 
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Comparison 

Difference 

T-value 

Significance 

Pictographs 

vs. 

Block Diagrams 

9.6 % 

1.536 

None 


Table 4b. Comparison of Results for System Status Display Formats. 


Even though the block diagram format was found to have a slightly 
higher comprehension rate than the pictograph format/ all test subjects 
stated that they preferred the pictograph format. One possible explanation 
for this discrepancy with the test scores is that a very limited amount of 
information was presented in the system status display formats. The 
information was so limited, in fact, that many of the subjects confessed 
that they were ignoring the meaning of the information until they filled 
out the questionnaire, and were instead relying on pattern recognition. 
Apparently simple pattern recognition was easier with the block diagram 
format, although this does not imply that the later interpretation of the 
pattern was either easier or more difficult than with the pictograph 
format. The time to fill out the questionnaire was not measured. 

An examination of the results of the different test subject groups 
(pilots vs. non-pilots) revealed that there was no significant difference 
between the pilots 1 and non-pilots 1 test scores for the pictograph format 
or the block diagram format. 

Fault History Display Formats (Experiment , 21 . ~ The tabulated results 

of the second experiment are shown in Tables 5a and 5b. The results show 
that the overall comprehension level was not quite as high as it was for 
the system status display formats: 80 percent for the pictograph frames 

format, 67 percent for the block diagram frames format, and 56 percent for 
the textual format. The results also show that the test scores for the 
pictograph frames format were on average 18 percent better than those for 
the block diagram frames format, and 34 percent better than those for the 
textual format. The test scores for the block diagram frames format were 
on average 17 percent better than those for the textual format. When 
statistical analysis was applied to these results, it was found that: (1) 

the difference between the pictograph frames format and the block diagram 
frames format was significant to the 2 percent level (less than a 2 percent 
chance existed that the difference was coincidental); (2) the difference 
between the pictograph frames format and the textual format was significant 
to the 0.5 percent level; and (3) the difference between the block diagram 
frames format and the textual format was not statistically significant. 

The t-test was again the statistical analysis technique applied. 
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Format 

Comprehension (Average Test Score) 

Pictograph Frames 

79.5 % 

Block Diagram Frames 

66.7 % 

Textual 

56.4 % 


Table 5a. Test Results of Fault History Display Format Experiment. 


Comparison 

Difference 

T-value 

Significance 

Pictograph Frames 
vs. 

Block Diagram Frames 

17.6% 

2.412 

2 % 

Pictograph Frames 
vs. 

Textual 

34.0 % 

4.017 

0.5 % 

Block Diagram Frames 
vs. 

Textual 

16.6 % 

1.656 

None 


Table 5b. Comparison of Results for Fault History Display Formats. 


The results of this experiment agreed much more closely with the 
expectations based on subject comments and reaction to the different 
formats. One possible explanation for the lower. performance on the block 
diagram frames format is that much more information was presented in the 
fault history display formats compared with the system status display 
formats. The increase in the amount of information prevented a simple 
pattern recognition approach, so the subjects had to interpret the 
information while the display was visible. An examination of the results 
of the different test subject groups (pilots vs. non-pilots) revealed that 
the non-pilots 1 test scores for the textual format were 43 percent better 
than the scores of the pilots for the same format. This difference in 
scores was found to be significant to the 1 percent level (T-value - 
2.993). There was no significant difference between the pilots 1 and non- 
pilots' test scores for the pictograph frames format or the block diagram 
frames format. 

One possible reason which may account for the difference in test 
scores for the textual format between the pilots and non-pilots is that 
when presented with information in the textual format, most of the pilots 
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expressed immediate dislike and may have consciously or unconsciously 
biased their performance against this format. The non-pilots as a group 
seemed to try to perform as well as possible on all formats. 


The responses to the first part of the subjective evaluation (see 
Appendix B) are summarized in Tables 6a and 6b. The Average Response (AR) 
for each question was calculated by the following formula: 

AR = SUM ( sum(a)*4 + sum(b)*3 + sum(c)*2 + sum(d)*l ) / (4*N) * 100% 

where: AR is the average response, 

sum (a) is the number of " (a) " responses given, 

sum (b) is the number of " (b) M responses given, etc., and 

N is the number of subjects. 

The possible responses represent a continuum of most favorable (a) to 
most unfavorable (d) reaction to each of the features. The average 
response for a feature is therefore similar to a grade point average. 

These responses indicate that: (1) the test subjects thought that the 

fault display interface program was relatively easy to operate (AR = 90 
percent); (2) the subjects preferred the pictograph format (AR =96 
percent) over the block diagram format (AR = 72 percent) for viewing system 
status information; (3) the subjects preferred the pictograph frames 
format (AR = 81 percent) over both the block diagram frames (AR = 62 
percent) and the textual (AR = 42 percent) formats for viewing fault 
history information, and preferred the block diagram frames format over the 
textual format; and (4) the subjects thought that displaying this type of 
fault history information was useful (AR = 73 percent) for understanding 
what has happened to a particular system. 


Question 

Average Response 

Overall Operation 

89.6 % 

Fault Detection: 


Pictographs 

95.8 % 

Block Diagrams 

71 .9 % 

Propagation Recognition: 


Pictograph Frames 

81.3% 

Block Diagram Frames 

61 .5 % 

Textual 

41 .7 % 

Usefulness of Fault History 

72.9 % 


Table 6a. Summary of Responses to the First Part of the Subjective 
Evaluation (see Appendix B) . 
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When statistical analysis was applied to these results, it was found 
that the 29 percent difference between the average responses to the 
pictograph format and the block diagram format was significant to the 0.5 
percent level. It was also found that: (1) the 28 percent difference 

between the average responses to the pictograph frames format and the block 
diagram frames format was significant to the 1 percent level; (2) the 64 
percent difference between the average responses to the pictograph frames 
format and the textual format was significant to the 0.5 percent level; and 
(3) the 38 percent difference in average responses to the block diagram 
frames format and the textual format was significant to the 5 percent 
level . 

An examination of the responses of the different test subject groups 
(pilots vs. non-pilots) revealed that no significant differences existed 
between the pilots 1 and non-pilots* average responses to any of the 
subjective evaluation questions regarding the display formats. 


Comparison 

Difference 

T-value 

Significance 

Pictographs 

vs. 

Block Diagrams 

28.6 % 

5.063 

0.5 % 

Pictograph Frames 
vs. 

Block Diagram Frames 

27.8 % 

3.035 

1 % 

Pictograph Frames 
vs. 

Textual 

64.4 % 

4.710 

0.5 % 

Block Diagram Frames 
vs. 

Textual 

38.4 % 

2.318 

5% 


Table 6b. Comparison of Responses from the First Part of the Subjective 
Evaluation . 


The responses to the second part of the subjective evaluation are 
given in Appendix B, and consist of the comments and suggestions given by 
the test subjects in response to the two questions in this part. Many of 
these suggestions were very specific to the formats being tested and the 
computer program presenting them, but several implied underlying concepts 
that should be explored in subsequent investigations. These concepts 
include: automatic selection of displays for systems in which a failure has 
occurred; inclusion of sensor readings, both pictorial and digital, 
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directly on the system status formats; and higher level notification of 
subsystem failures to show their effect on overall aircraft capability. 

The responses to the third part of the subjective evaluation (see 
Appendix B) are summarized in Table 7. The score for each display format 
characteristic was calculated by the following formula: 

SCORE = sum ( 1 ) * 3 + sum(2)*2 + sum(3)*l - sum(4)*l - sum(5)*2 - sum(6)*3 

where: SCORE is the ranking of each item (a) through (v) in question 10, 
sum (1) is the number of "1" rankings received, 
sum(2) is the number of "2" rankings received, etc. 


The possible rankings represent what each test subject thought were 
the three best (1-best liked, 2, 3) and the three worst (4,5,6-least liked) 
features of the display concepts. Possible scores range from 36 down to 
-36. 


Item Description 

Score 

Amount of information presented 

4 

Use of color 

1 5 

Size of alphanumerics 

0 

Size of system diagrams 

3 

Size of history frames 

-1 

System display format 

8 

Inclusion of labels on block diagrams 

-3 

Deletion of labels on pictographs 

0 

History display format 

-6 

Menu format 

0 

Illumination of menu option for changes 

12 

Ease of switching between display types 

-2 

Ability to select current system history only 

-2 

Chronological arrangement of history frames 

0 

Lack of history info updates 

-14 

Ordering of history frame pages 

-9 

Text formats 

-12 

Use of pictographs 

9 

Block diagram formats 

-2 

Absence of flashing menu option 

-2 


Table 7. Summary of Rankings from the Third Part of the Subjective 
Evaluation (see Appendix B) . 


The responses indicate that the test subjects were slightly favorable 
towards the amount of information presented (SCORE = 4), very favorable 
towards the use of color (SCORE = 15), neutral towards the size of the 
alphanumerics (SCORE - 0), slightly favorable towards the size of the 
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system diagrams (SCORE = 3), and slightly negative towards the size of the 
history frames (SCORE = -1). The subjects were also favorable to the 
system display format arrangement (SCORE » 8) , slightly negative towards 
the inclusion of labels on the block diagrams (SCORE * -3) , neutral on the 
deletion of labels on the pictographs (SCORE = 0) , and negative towards the 
history display format arrangement (SCORE = -6) . The subjects were neutral 
towards the menu format arrangement (SCORE = 0), very favorable towards the 
illumination of menu options for changes in status (SCORE = 12) , slightly 
negative towards the ease of switching between display types (SCORE = -2) , 
and slightly negative towards the ability to select the history of the 
currently viewed system only (SCORE = -2) . The subjects were also neutral 
towards the chronological arrangement of the history frames (SCORE - 0) , 
very negative towards the lack of information updates while viewing a 
system history (SCORE = -14), and negative towards the ordering of the 
history frame pages (SCORE = -9) . 

In addition to the listed display format characteristics, several 
characteristics were included by test subjects: the responding subjects 

were very negative towards the use of text formats (SCORE = -12), favorable 
towards pictograph formats (SCORE = 9) , slightly negative towards block 
diagram formats (SCORE * -2), and slightly negative towards the absence of 
a flashing menu option for status change notification (SCORE ~ -2) . 

To summarize, the responses to the third part of the subjective 
evaluation indicated that the test subjects liked the pictograph formats 
best, liked the use of color for differentiating component status, and 
liked a form of notification when changes in system status occurred. 
However, the subjects did not particularly like any of the history formats, 
did not like the lack of status updates while viewing a system history, and 
did not like the ordering (oldest first) of the pages of history 
information. Based on these responses, it is clear that additional 
research is necessary to examine alternatives for displaying fault history 
(propagation) information. 


CONCLUDING REMARKS 


The results of the experiments suggest that pictographs should be used 
over both block diagrams and text for system status and fault history 
display formats. The responses to the first part of the subjective 
evaluation agree with these test results: the subjects stated that they 
preferred pictographs over block diagrams and text for viewing system 
status and fault history information. The responses to the second and 
third parts of the subjective evaluation also agree with the test results, 
but indicate that much additional research is necessary to resolve specific 
format and interaction problems. 
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APPENDIX A 
TEST MATERIALS 



BACKGROUND INFORMATION 


TEST SUBJECT 


1 . Name 


2. Are you a pilot? Yes No 


3. If yes, how many total flight hours do you have? 


4. Please list all ratings 


5. Have you ever flown a flight simulator? 
Yes No 


6. If yes, give approximate hours and simulator type. 


7. Even if you are not a pilot and have never flown a flight 

simulator before, are you familiar with basic aircraft systems 
and how they operate? 

Yes No 
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QUESTIONNAIRE PRIOR TO TESTS 
TEST SUBJECT 


In general, what would you want an automated fault diagnosis 
system to tell you when it detects that something has gone 
wrong? 


How would you want it to present this information to you? 


What would you want an automated fault diagnosis system to 
show you so that you could trace the fault history of a 
particular system? 


How would you want it to present this fault history informa- 
tion to you? 



FLASH TEST NUMBER 


TEST SUBJECT 


Name the component (s) of the left engine that were failed. 


Name the component (s) of the left engine that were partially- 
failed. 


Name the component (s) of the left engine that were okay. 


Name the component (s) of the right engine that were failed. 


Name the component (s) of the right engine that were partially 
failed. 


Name the component (s) of the right engine that were okay. 



(HISTORY) FLASH TEST NUMBER 
TEST SUBJECT 


In which engine did a fault first occur? 


Which component (s) in that engine first indicated a fault? 


Did the initial fault (s) in that engine remain throughout the 
viewed time interval? 


Which direction (towards the inlet or towards the outlet) in 
the engine did the fault (s) seem to propagate? [State N/A if 
the fault (s) did not seem to propagate.] 


Which component (s) in the other engine first indicated a fault? 
[State N/A if no faults were indicated.] 


Did the initial fault (s) in this engine remain throughout the 
viewed time interval? [State N/A if no faults were indicated.] 


Which direction (towards the inlet or towards the outlet) in 
the engine did the fault (s) seem to propagate? [State N/A 
if no faults were indicated or if the fault (s) did not seem 
to propagate.] 


SUBJECTIVE EVALUATION 


TEST SUBJECT 


How would you rate the overall ease of operating the display 
program? 

(a) very easy to operate 

(b) somewhat easy to operate 

(c) somewhat difficult to operate 

(d) very difficult to operate 


How easy was it for you to detect the faulted subsystems when 
the information was presented in pictograph form? 

(a) very easy to detect 

(b) somewhat easy to detect 

(c) somewhat difficult to detect 

(d) very difficult to detect 


How easy was it for you to detect the faulted subsystems when 
the information was presented in block diagram form? 

(a) very easy to detect 

(b) somewhat easy to detect 

(c) somewhat difficult to detect 

(d) very difficult to detect 


How easy was it for you to recognize the sequence of subsystem 
faults when the fault history information was presented in the 
form of four miniature system displays using pictographs? 

(a) very easy to recognize 

(b) somewhat easy to recognize 

(c) somewhat difficult to recognize 

(d) very difficult to recognize 


How easy was it for you to recognize the sequence of subsystem 
faults when the fault history information was presented in the 
form of four miniature system displays using block diagrams? 

(a) very easy to recognize 

(b) somewhat easy to recognize 

(c) somewhat difficult to recognize 

(d) very difficult to recognize 



6. How easy was it for you to recognize the sequence of subsystem 
faults when the fault history information was presented in the 
form of a word list? 

(a) very easy to recognize 

(b) somewhat easy to recognize 

(c) somewhat difficult to recognize 

(d) very difficult to recognize 


7. In general, how useful did you find the type of fault history 
information, presented in the three different forms, to be in 
understanding what has happened to a particular system? 

(a) very useful 

(b) somewhat useful 

(c) somewhat useless 

(d) very useless 


8. How would you improve the screen formats of the display 
concepts? 


9. How would you improve the menu option selection devices of 
the display concepts? 
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10. Please rank the three things you liked best about the display 
concepts (l=best liked, 2, 3), and the three things you liked 
least about the display concepts <4,5,6=least liked). 

(a) total amount of information presented 

(b) use of color 

(c) size of alphanumerics 

(d) size of system diagrams 

(e) size of history frames 

(f) system display format 

(g) inclusion of labels on block diagrams 

(h) deletion of labels on pictographs 

(i) history display format 

(j) menu format 

(k) illumination of menu option for system with status 
change 

(1) ease of switching back and forth between system and 

history displays 

(m) ability to select the history display for the currently 

viewed system only 

(n) chronological arrangement of history frames on each 
page of history information 

(o) lack of updating history information while viewing 
the history display 

(p) order in which pages of history information are 

presented 

(q) other __ : 

(r) other 

(s) other _____ — 

(t) other 

(u) other 

(v) other 
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APPENDIX B 

SUMMARY OF RESPONSES TO SUBJECTIVE EVALUATION 
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For questions 1 - 7 of the subjective evaluation, the number of subjects 
selecting each option (a) - (d) is given in square brackets next to that 
option . 


1. How would you rate the overall ease of operating the display 
program? 


(a) very easy to operate [7] 

(b) somewhat easy to operate [5] 

(c) somewhat difficult to operate [0] 

(d) very difficult to operate [0] 


2 . How easy was it for you to detect the faulted subsystems when 
the information was presented in pictograph form? 


(a) very easy to detect [10] 

(b) somewhat easy to detect [2] 

(c) somewhat difficult to detect [0] 

(d) very difficult to detect [0] 


3. How easy was it for you to detect the faulted subsystems when 
the information was presented in block diagram form? 


(a) very easy to detect [1] 

(b) somewhat easy to detect [8.5] 

(c) somewhat difficult to detect [2.5] 

(d) very difficult to detect [0] 


4. How easy was it for you to recognize the sequence of subsystem 
faults when the fault history information was presented in the 
form of four miniature system displays using pictographs? 


(a) very easy to recognize [4] 

(b) somewhat easy to recognize [7] 

(c) somewhat difficult to recognize [1] 

(d) very difficult to recognize [0] 


5. How easy was it for you to recognize the sequence of subsystem 
faults when the fault history information was presented in the 
form of four miniature system displays using block diagrams? 


(a) very easy to recognize [0] 

(b) somewhat easy to recognize [6.5] 

(c) somewhat difficult to recognize [4.5] 

(d) very difficult to recognize [1] 
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6 . 


How easy was it for you to recognize the sequence of subsystem 
faults when the fault history information was presented in the 
form of a word list? 


(a) very easy to recognize [1] 

(b) somewhat easy to recognize [1] 

(c) somewhat difficult to recognize [3] 

(d) very difficult to recognize [7] 


7. In general / how useful did you find the type of fault history 
information, presented in the three different forms, to be in 
understanding what has happened to a particular system? 


(a) 

very useful 

[4] 

(b) 

somewhat useful 

[4] 

<c) 

somewhat useless 

[3] 

<d) 

very useless 

[1] 


For question 8 and 9 of the subjective evaluation, a condensed version 
of frequent responses is given below each question. 


8 . How would you improve the screen formats of the display 
concepts? 

Would like to see the partial faults and full failures displayed 
separately when the distinction berween the two is needed. 

Use pictographs, add information on screen to indiate why a system is 
failed (i.e. draw an oil pressure gage showing low pressure). 

Use pictographs, and show a fire in the combustor to indicate operation 
of the engine . 

Separate the components on the pictorial display - easier to 
distinguish. 

Make the picture larger. 

Rotate block diagram format so they are vertical. Include some 
digitally displayed information pertaining to system performance. 

Rotate block diagrams 90 degrees, and put #1 engine on left side. 

Show component that caused the status change by flashing it or 
something. Only scroll 2 frames instead of 4 in the history. 

Top level display should show all major 'systems {engine, control 
surfaces, etc.) . If the engine fails, this should show up on this 
display, then you can go to the engine display for details. 

Flash most recent and/or unacknowledged failures. 
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9. How would you improve the menu option selection devices of 
the display concepts? 

Should experiment with automatically selecting the appropriate display 
when a failure occurs. 

Use a touch panel for selecting menu options. 

Consider flashing lights for the menu options. 

Allow greater selection capability; only use one history page, and bump 
off old frames. 

Use a touch panel display. 

Use a touch panel with dedicated menu boxes, like line select CDUs . 

The color used to light up the menu option should reflect the severity 
of the change that occurred. 

The menus should be logical subsets, and function buttons should 
indicate which screen you are currently viewing. A separate acknowledge 
button might be useful. 


For question 10 of the subjective evaluation, the number of subjects 
assigning each value 1 - 6 to the options (a) - (p) is given- in square 
brackets underneath each option. 


10 Please rank the three things you liked best about the display 
concepts (1-best liked, 2, 3), and the three things you liked 
least about the display concepts (4,5,6=least liked). 


(a) total amount of information presented 


1 [1] 2 [0] 3 [2] 

4 

[i] 

5 

[0] 

6 

[0] 

(b) use of color 
1 [3] 2 [2] 3 [2] 

4 

[0] 

5 

[0] 

6 

[0] 

(c) size of alphanumerics _ 
1 [0] 2 [0] 3 [0] 

4 

[0] 

5 

[0] 

6 

[0] 

(d) size of system diagrams 
1 [0] 2 [1] 3 [1] 

4 

[0] 

5 

[0] 

6 

[0] 

(e) size of history frames 
1 [0] 2 [0] 3 [0] 

4 

[1] 

5 

[0] 

6 

[0] 

(f) system display format 
1 [1] 2 [2] 3 [1] 

4 

[0] 

5 

[0] 

6 

[0] 

(g) inclusion of labels on 
1 [0] 2 [0] 3 [0] 

block 

4 

; diagrams 
[0] 5 

[0] 

6 

[1] 
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5 [0] 


6 [ 0 ] 


(h) deletion of labels on pictographs 
1 [0] 2 [0] 3 [1] 4 [1] 


1 [ 0 ] 


2 [ 0 ] 


3 [0] 

i — i 

o 

5 [0] 

6 [2] 

3 [1] 

4 [1] 

5 [0] 

6 [0] 


(j) menu format 

1 [ 0 ] 2 [ 0 ] 

(k) illumination of menu option for system with status 
change 

1 [3] 2 [2] 3 [0] 4 [1] 5 [0] 6 [0] 

(l) ease of switching back and forth between system and 
history displays 

1 [0] 2 [1] 3 [0] 4 [0] 5 [2] 6 [0] 

(m) ability to select the history display for the currently 
viewed system only 

1 [0] 2 [2] 3 [1] 4 [1] 5 [O'] 6 [2] 


(n) chronological arrangement of history frames on each 

page of history information 

1 [0] 2 [0] 3 [2] 4 [2] 5 [0] 6 [0] 

(o) lack of updating history information while viewing 
the history display 

1 [0] 2 [0] 3 [O'] 4 [3] 5 [4] 6 [1] 

(p) order in which pages of history information are 
presented 

1 [0] 2 [0] 3 [0] 4 [0] 5 [3] 6 [1] 


The following four options (q) - (t) were included by several test 
subjects. The number of subjects assigning each value 1 - 6 to these 
additional options is also given in square brackets underneath each 
opt ion . 


(q) text formats 


1 [0] 2 [0] 

3 [0] 

4 [0] 

5 [0] 

6 [4] 

(r) pictograph formats 




1 [3] 2 [0] 

3 [0] 

4 tor 

5 [0] 

6 [0] 

(s) block diagram 

formats 




1 [0] 2 [0] 

3 [0] 

4 [0] 

5 [1] 

6 [0] 

(t) steady (rather 

than flashing) menu 

option 


1 [0] 2 [0] 

3 [0] 

4 [0] 

5 [1] 

6 [0] 
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